This PR allows bench to have a data_processing stanza, currently we have two styles

1. a functional recipe style, where the formatting functions need to be implemented in python.
2. a jinja style, where the formatting is given in a template

Loss Masking

• loss masking is done automatically if --response_template is passed into the arguments.

Older Style

In this style, the recipe is specified by the formatting flag. This requires updating python code to understand
what needs to be done for that particular recipe

• also for the recipe to understand the fields of the dataset, you need to specify input_field and so on
• this style is not so good, as everything is opaque. It requires opening the data processing functions to know exactly what processing is happening.

data_processing:
  dataset_name: yahma/alpaca-cleaned
  formatting: "instruct"
  tokenize: True
  input_field: input

Chat Templates

In this style we rely on HF's integration of chat templating.

• this is more flexible and the preferred approach.

data_processing:
  dataset_name: yahma/alpaca-cleaned
  chat_template: |
    {%- for message in messages %}
        {% if message['input'] != '' %}
    Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

        {% else %}
    Below is an instruction that describes a task. Write a response that appropriately completes the request.

        {% endif %}
    ### Instruction:
    {{ message['instruction'] }}

        {% if message['input'] != '' %}
    ### Input:
    {{ message['input'] }}

        {% endif %}
    ### Response:
    {{ message['output'] + eos_token }}
    {% endfor %}
  tokenize: True